https://vapi.ai/ logo
Join Discord
Powered by
Custom LLM + tool calls = stream chunks are gettin...
# support
i
We have a Custom LLM implementation that VAPI calls over an SSE connection. We want this sequence: 1. POST /vapi/llm to start 2. agent generate sequence 3. tool call happens, we speak "hang on let me check that for you" 4. tool call occurs 5. VAPI speaks response to user When I test the endpoint locally, I can see the 
data: {"id": ...}
chunks being streamed through fine. However, on the voice side, VAPI just queues all of the chunks up and speaks them together. Which sorta ruins the point of the waiting message. How do you get VAPI to speak at designated chunks rather than wait for the entire stream to finish? Conversely, I've tried using the controlClient to 
say
a response when the tool starts, but VAPI breaks the connection when this happens.
v
To have VAPI speak interim messages (like "hang on let me check that for you") before a tool call, you must send a complete SSE event for that message and then end the message before triggering the tool call. For example, stream your interim message as a full SSE event, then send the tool call event as a separate message:
Copy code
js
// Stream interim message
res.write('data: {"id":"stream_1","path":"chat.output[0].content","delta":"Hang on let me check that for you."}\n\n');
// End interim message so VAPI speaks it
res.write('data: [DONE]\n\n');
// Now trigger the tool call event
// ...send tool call payload...
This ensures VAPI speaks the interim message immediately, then waits for the tool call result before continuing. See the [Custom LLM Tool Calling Integration documentation](https://docs.vapi.ai/customization/tool-calling-integration) for more details. Source: - [Custom LLM Tool Calling Integration documentation](https://docs.vapi.ai/customization/tool-calling-integration)
i
@Vapi Support Bot I have already tried everything in there. Am I supposed to use data: [DONE] twice? once for the first message and then after the agent has completed the tool calls and prepared a response?
v
Yes, you should use 
data: [DONE]
twice—once after the interim message and again after the final response. This signals Vapi to speak each message segment immediately. Here’s a concrete example:
Copy code
js
// 1. Stream interim message
res.write('data: {"id":"stream_1","path":"chat.output[0].content","delta":"Hang on let me check that for you."}\n\n');
res.write('data: [DONE]\n\n'); // Ends interim message

// 2. After tool call, stream final response
res.write('data: {"id":"stream_2","path":"chat.output[0].content","delta":"Here is the result of your request."}\n\n');
res.write('data: [DONE]\n\n'); // Ends final message
This ensures Vapi speaks each part at the right time. Reference: [Custom LLM Tool Calling Integration documentation](https://docs.vapi.ai/customization/tool-calling-integration) Source: - [Custom LLM Tool Calling Integration documentation](https://docs.vapi.ai/customization/tool-calling-integration)
i
@Vapi Support Bot it would be a good idea to add this example to the docs
v
That's a great suggestion! Including a concrete example like the one above would definitely help clarify the correct streaming sequence for others working with custom LLMs and tool calls in Vapi. If you have any other feedback or ideas for improving the process, feel free to share!
i
@Vapi Support Bot that doesn't work only the first part gets spoken
v
It appears you've implemented the documented streaming pattern (
data: [DONE]
after each message), but only the first part is spoken—this is not addressed in the official documentation. Since the expected approach isn't working, I recommend reaching out to [Vapi Support](https://docs.vapi.ai/support) with your implementation details and logs for further troubleshooting. There may be an internal issue or undocumented limitation affecting your use case. Source: - [Custom LLM Tool Calling Integration documentation](https://docs.vapi.ai/customization/tool-calling-integration)
c
Hi Ian, The core problem is that your Custom LLM SSE stream does not match the format VAPI expects. VAPI’s Custom LLM implementation strictly follows the standard OpenAI streaming format, not a custom SSE format with 
path
fields or multiple 
[DONE]
markers. What’s going wrong - After the first 
data: [DONE]
, VAPI considers the entire LLM response fully completed and closes the stream. - Because of this, you cannot send multiple 
[DONE]
messages in a single Custom LLM request. - This is why only the first spoken segment is heard — everything after the first 
[DONE]
is ignored. Recommendation: Use the 
<flush />
tag in a single stream This is the supported and reliable way to force VAPI to speak interim content immediately without ending the stream. Example (standard OpenAI streaming format):
Copy code
js
// Interim message — spoken immediately
res.write(
  'data: {"choices":[{"delta":{"content":"Hang on, let me check that for you.<flush />"}}]}\n\n'
);

// Tool call
res.write(
  'data: {"choices":[{"delta":{"tool_calls":[{"id":"call_123","type":"function","function":{"name":"search","arguments":"{}"}}]}}]}\n\n'
);

// Final spoken response
res.write(
  'data: {"choices":[{"delta":{"content":"Here is the result of your request."}}]}\n\n'
);

// End the stream ONCE
res.write('data: [DONE]\n\n');
Key takeaways - Use standard OpenAI streaming JSON (
choices[].delta
) - Do not send multiple 
[DONE]
events - Use 
<flush />
to control when VAPI speaks - End the stream exactly once This should resolve the issue where only the first chunk is spoken.
i
Hey Kyle - In practice I'm finding that your suggestion is not how Vapi performs. > After the first data: [DONE], VAPI considers the entire LLM response fully completed and closes the stream. Yes this is what I assumed, but your AI response is wrong. > This is why only the first spoken segment is heard — everything after the first [DONE] is ignored. I started out by just doing a single DONE at the end of the stream, but Vapi still doesn't spean the messages as they come in. Vapi will NOT speak any messages until it receives done and then speaks all of them, so it sounds like "Let me check that for you I couldn't find anything in the knowledgebase"
I will try the command lets see if that does it.
still seeing same behavior. The "One moment — let me check that for you." and then the "I found ..." come right after another. Vapi is not speaking when it receives stream chunks
2 Views
PreviousNext
Powered by